CPU and Max RSS Analysis tools #6663

ChrisPaulBennett · 2025-03-12T09:14:38Z

This apart of 3 pull requests for adding CPU time and Max RSS analysis to the Cylc UI.

This adds the Max RSS and CPU time (as measured by cgroups) to the table view, box plot and time series views.

This adds a python profiler script. This profiler will will be ran by cylc in the same crgroup as the cylc task. It will periodically poll cgroups and save data to a file. Cylc will then store these values in the sql db file.

Linked to;
cylc/cylc-ui#2100
cylc/cylc-uiserver#675

Check List

I have read CONTRIBUTING.md and added my name as a Code Contributor.
Contains logically grouped changes (else tidy your branch by rebase).
Does not contain off-topic changes (use other PRs for other changes).
Applied any dependency changes to both setup.cfg (and conda-environment.yml if present).
Tests are included (or explain why tests are not needed).
Changelog entry included if this is a change that can affect users
Cylc-Doc pull request opened if required at cylc/cylc-doc/pull/XXXX.
If this is a bug fix, PR should be raised against the relevant ?.?.x branch.

oliver-sanders

🎉

cylc/flow/etc/job.sh

cylc/flow/job_file.py

cylc/flow/etc/job.sh

cylc/flow/scripts/profile.py

tests/functional/jobscript/02-profiler.t

Changed the name of the profiler module. Linting Profiler sends KB instead of bytes Time Series now working CPU/Memory Logging working

Initial profiler implementation (non working) Changed the name of the profiler module. Linting Profiler sends KB instead of bytes Time Series now working CPU/Memory Logging working Adding profiler unit tests updating tests Fail gracefully if cgroups cannot be found Revert "Fail gracefully if cgroups cannot be found" This reverts commit 92e1e11c9b392b4742501d399f191f590814e95e. Linting Modifying unit tests Linting Changed the name of the profiler module. Profiler sends KB instead of bytes Time Series now working

Co-authored-by: Hilary James Oliver <[email protected]>

* The COPYING file appears to have moved from `dist-info/COPYING` into `dist-info/licenses/COPYING`.

Co-authored-by: Hilary James Oliver <[email protected]>

* Attempt to fix flaky test. * Cut out shell profile files to omit some spurious stderr.

Co-authored-by: Hilary James Oliver <[email protected]>

* Attempt to fix flaky test. * Cut out shell profile files to omit some spurious stderr.

Revert "Fail gracefully if cgroups cannot be found" This reverts commit 92e1e11c9b392b4742501d399f191f590814e95e. Adding profiler unit tests updating tests

cylc/flow/etc/job.sh

cylc/flow/scripts/profiler.py

oliver-sanders · 2025-10-21T11:00:54Z

We seem to have agreement on the validity cgroups field(s) we're polling, we should be able to push forward with this quickly now.

Last couple of outstanding comments:

Remove global variables.
- Information is being duplicated into both the Process object and the global variables.
- We can remove the global variables in favour of the Process object.
- POC refactor: https://github.com/oliver-sanders/cylc-flow/pull/new/cylc_profiler.remove_global_variables
  - Add a cgroup_version field to the Process object.
  - Pass the Process object into the stop_profiler method.
  - Remove usage of global variables.
Backup the profiler output to the job.status file.
- This PR is using the cylc.flow.send_messages interface to send the message back to the scheduler.
- However, we should probably be using the record_messages interface (MB I think, sry).
- This interface will additionally write the message to the job.status file.
- This way, the message will not be lost if the scheduler is stopped at the time the message is sent, or if a network issue prevents the transmission of the message.
- Note, this interface does not presently support a comms_timeout option. This isn't a biggie, we can drop this.

Then we can get Dave to ok the cgroup stuff and we're away...

cylc/flow/scripts/profiler.py

ChrisPaulBennett · 2025-10-21T14:26:16Z

* Pass the `Process` object into the `stop_profiler` method.

How do I do that? I couldn't see a way to do it. Since I'm not calling the function, its the registered function for sigkill
The use of globals is the only way I could see to get around. I'd love to get rid of them.

*EDIT. Sorry, I didn't see the pull request, I'll go through it now

ChrisPaulBennett marked this pull request as draft March 12, 2025 09:19

oliver-sanders reviewed Mar 12, 2025

View reviewed changes

oliver-sanders added this to the 8.x milestone Mar 12, 2025

oliver-sanders assigned ChrisPaulBennett Mar 12, 2025

This was referenced Mar 13, 2025

CPU and Max RSS Analysis tools cylc/cylc-ui#2100

Open

CPU and Max RSS Analysis tools cylc/cylc-uiserver#675

Open

ChrisPaulBennett force-pushed the cylc_profiler branch 2 times, most recently from fb1b12b to c5d30b3 Compare March 21, 2025 11:37

Initial profiler implementation (non working)

246a4c1

Changed the name of the profiler module. Linting Profiler sends KB instead of bytes Time Series now working CPU/Memory Logging working

ChrisPaulBennett force-pushed the cylc_profiler branch 2 times, most recently from 4a6dbe8 to 30a7bb0 Compare April 1, 2025 15:31

ChrisPaulBennett and others added 19 commits April 2, 2025 09:34

GH Actions: use explicit bash shell & other defaults

171f7ee

host-select: fix compatibility with force-condemned hosts

7d50d0b

Update cylc/flow/cfgspec/globalcfg.py

8608088

Co-authored-by: Hilary James Oliver <[email protected]>

actions: update build action to support python 3.13

060056b

* The COPYING file appears to have moved from `dist-info/COPYING` into `dist-info/licenses/COPYING`.

host-select: fix compatibility with force-condemned hosts

a0cdcd6

Update cylc/flow/cfgspec/globalcfg.py

3d4a187

Co-authored-by: Hilary James Oliver <[email protected]>

tests: convert unittest to pytest

3057edd

tests/u: test_subprocpool.py::test_run_command_writes_to_err

a6ecd29

* Attempt to fix flaky test. * Cut out shell profile files to omit some spurious stderr.

Time Series now working

e5119b4

Profiler sends KB instead of bytes

9f593fc

Modifying unit tests

5f84c2f

host-select: fix compatibility with force-condemned hosts

47a4a98

Update cylc/flow/cfgspec/globalcfg.py

356abff

Co-authored-by: Hilary James Oliver <[email protected]>

tests: convert unittest to pytest

6ed1770

tests/u: test_subprocpool.py::test_run_command_writes_to_err

4cfcffe

* Attempt to fix flaky test. * Cut out shell profile files to omit some spurious stderr.

Linting

1e5b804

Fail gracefully if cgroups cannot be found

38c31f5

Revert "Fail gracefully if cgroups cannot be found" This reverts commit 92e1e11c9b392b4742501d399f191f590814e95e. Adding profiler unit tests updating tests

tests: convert unittest to pytest

339af9c

ChrisPaulBennett added 7 commits August 5, 2025 15:15

testing macos

2bc9ec6

testing macos

8ca760d

testing macos

ebbb8f0

testing macos

6f65ebd

testing macos

54553c1

testing macos

556f3ba

testing macos

f70c1d3

oliver-sanders reviewed Aug 18, 2025

View reviewed changes

cylc/flow/etc/job.sh Outdated Show resolved Hide resolved

oliver-sanders reviewed Aug 18, 2025

View reviewed changes

ChrisPaulBennett and others added 4 commits October 20, 2025 15:11

change how profiler is killed to be POSIX compliant

1e93434

Added recording of memory allocation

797b8ae

Updating unit tests

5bfaf05

profiler: remove global variables

3c32069

dpmatthews requested changes Oct 21, 2025

View reviewed changes

cylc/flow/scripts/profiler.py Outdated Show resolved Hide resolved

cylc/flow/scripts/profiler.py Outdated Show resolved Hide resolved

ChrisPaulBennett added 14 commits October 23, 2025 14:06

Refactored to global variables

8592107

Code review changes

6c95914

Linting

e30a1a4

Linting

c52f7fb

Updating unit tests

19ba481

Linting

48bb9c4

Adding unit test coverage

fb0a156

Adding unit test coverage

ba94659

Adding unit test coverage

ed7f4fc

Adding unit test coverage

5de1e27

Linting

2762d56

Linting

e1978c1

Unit test coverage

61f0ceb

Linting

2b27c25

CPU and Max RSS Analysis tools #6663

Are you sure you want to change the base?

CPU and Max RSS Analysis tools #6663

Uh oh!

Conversation

ChrisPaulBennett commented Mar 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

oliver-sanders left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

oliver-sanders commented Oct 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ChrisPaulBennett commented Oct 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ChrisPaulBennett commented Mar 12, 2025 •

edited

Loading

oliver-sanders commented Oct 21, 2025 •

edited

Loading

ChrisPaulBennett commented Oct 21, 2025 •

edited

Loading